Tone mapping for low-light video frame enhancement

ABSTRACT

A technique is provided for generating sharp, well-exposed, color images from low-light images. A series of under-exposed images is acquired. A mean image is computed and a sum image is generated each based on the series of under-exposed images. Chrominance variables of pixels of the mean image are mapped to chrominance variables of pixels of the sum image. Chrominance values of pixels within the series of under-exposed images are replaced with chrominance values of the sum image. A set of sharp, well-exposed, color images is generated based on the series of under-exposed images with replaced chrominance values.

RELATED APPLICATIONS

This application is a Continuation-in Part (CIP) of U.S. patentapplication Ser. No. 12/485,316, filed Jun. 16, 2009, now U.S. Pat. No.8,199,222, which is a CIP of Ser. No. 12/330,719, filed Dec. 9, 2008,now U.S. Pat. No. 8,264,576, which is a CIP of U.S. Ser. No. 11/856,721,filed Sep. 18, 2007, now U.S. Pat. No. 8,417,055, which claims priorityto U.S. provisional application No. 60/893,116, filed Mar. 5, 2007. Thisapplication is also related to U.S. Ser. No. 12/336,416, filed Dec. 16,2008; and U.S. Ser. No. 11/753,098, filed May 24, 2007; and U.S. Ser.No. 12/116,140, filed May 6, 2008. All of these related applicationsbelong to the same assignee and are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing method andapparatus, and specifically to low-light video frame enhancement.

2. Description of the Related Art

Sensor arrays for digital cameras exist for capturing color photos.Sensors known as RGBW sensors are used for capturing red, green, andblue colors, and for capturing luminance information for multiple pixelsof an array of pixels. The red, green and blue pixels include filterssuch that only certain narrow ranges of wavelengths of incident lightare counted. The white pixels capture light of red, green and bluewavelengths, i.e., of a broader range of wavelengths than any of theblue, green and red pixels. Thus, the white pixels are typicallybrighter than any of the blue, red and green pixels if they are exposedfor the same duration. Noise removal algorithms can tend to blur faceregions in an undesirable manner. Noise removal algorithms are describedat U.S. patent application Ser. Nos. 11/856,721 and 11/861,257, whichare hereby incorporated by reference, as are U.S. Ser. Nos. 10/985,650,11/573,713, 11/421,027, 11/673,560, 11/319,766, 11/744,020, 11/753,098,11/752,925, and 12/137,113, which are assigned to the same assignee asthe present application and are hereby incorporated by reference.

Kodak has developed a RGBW color filter pattern differing from thepreviously known Bayer Color Filter. The RGBW pattern of Kodak isreferred to as a Color Filter Array (CFA) 2.0. One half of cells in aRGBW pattern are panchromatic, i.e. sensing all of the color spectrum (Ycomponent)—usually called white cells. This way more light energy isaccumulated in the same amount of time than for color pixels. A Bayerfilter uses only ⅓ (˜0.33) of color spectrum energy. An RGBW filter uses4/6 (˜0.67) of the energy, where ½ comes from white cells and ⅙ from RGBcells.

CFA Array looks something like the following:

-   -   WBWG . . .    -   BWGW . . .    -   WGWR . . .    -   RWRW . . .

In this context, the following are incorporated by reference: U.S. Pat.Nos. 7,195,848, 7,180,238, 7,160,573, 7,019,331, 6,863,368, 6,607,873,6,602,656, 6,599,668, 6,555,278, 6,387,577, 6,365,304, 6,330,029,6,326,108, 6,297,071, 6,114,075, 5,981,112, 5,889,554, 5,889,277,5,756,240, 5,756,239, 5,747,199, 5,686,383, 5,599,766, 5,510,215,5,374,956, and 5,251,019.

Two source images nominally of the same scene may be used to produce asingle target image of better quality or higher resolution than eitherof the source images.

In super-resolution, multiple differently exposed lower resolutionimages can be combined to produce a single high resolution image of ascene, for example, as disclosed in “High-Resolution ImageReconstruction from Multiple Differently Exposed Images”, Gunturk etal., IEEE Signal Processing Letters, Vol. 13, No. 4, April 2006; or“Optimizing and Learning for Super-resolution”, Lyndsey Pickup et al,BMVC 2006, 4-7 Sep. 2006, Edinburgh, UK, which are hereby incorporatedby reference. However, in super-resolution, blurring of the individualsource images either because of camera or subject motion are usually notof concern before the combination of the source images.

U.S. Pat. No. 7,072,525, incorporated by reference, discloses adaptivefiltering of a target version of an image that has been produced byprocessing an original version of the image to mitigate the effects ofprocessing including adaptive gain noise, up-sampling artifacts orcompression artifacts.

US published applications 2006/0098890, 2007/0058073, 2006/0098237,2006/0098891, European patent EP1779322B1, and PCT Application No.PCT/EP2005/011011 (WO/2006/050782), which are each hereby incorporatedby reference, describe uses of information from one or morepresumed-sharp, short exposure time (SET) preview images to calculate amotion function for a fully exposed higher resolution main image toassist in the de-blurring of the main image.

Indeed many other documents, including US 2006/0187308, Suk Hwan Lim etal.; and “Image Deblurring with Blurred/Noisy Image Pairs”, Lu Yuan etal, SIGGRAPH07, Aug. 5-9, 2007, San Diego, Calif., which areincorporated by reference, are directed towards attempting to calculatea blur function in the main image using a second reference image beforede-blurring the main image.

Other approaches, such as may be disclosed in US2006/0017837, which isincorporated by reference, involve selecting information from two ormore images, having varying exposure times, to reconstruct a targetimage where image information is selected from zones with high imagedetails in SET images and from zones with low image details in longerexposure time images.

Nowadays, even though image processing techniques have evolved, stilllow-light scenes tend to lead to very dark frames in videos. By thesemeans the visual quality is low. For example, movies taken in low-lightconditions (e.g. 30 lux illumination), at 30 fps, generally result invery dark, underexposed frames that have a very low contrast, so objectsare not distinguishable. Tone mapping (amplification) algorithms may beapplied to improve such images. Although these may provide a betterexposed image, there are difficulties in restoring the natural colors ofthe scene. For example, simple summation or forcing higher exposuretends to lead to frames degraded by motion blur.

United States published patent application no. 2009/0303343, whichbelongs to the same assignee and is incorporated by reference, describesbackground techniques generally involving two different images acquiredeither at different moments of time or by two different sensors. It isdesired to have a new technique that utilizes images having identical orsubstantially similar content, and such is provided herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 schematically illustrates conventional video frame acquisition.

FIG. 2 schematically illustrates video frame enhancement using luminanceand/or color data from a well-exposed image, and applying the data to arelatively less exposed, sharper image in accordance with a firstembodiment.

FIG. 3 schematically illustrates video frame enhancement using luminanceand/or color data from a well-exposed image, and applying the data to arelatively less exposed, sharper image in accordance with a secondembodiment.

FIG. 4 schematically illustrates video frame enhancement using luminanceand/or color data from a well-exposed image, and applying the data to arelatively less exposed, sharper image in accordance with a thirdembodiment.

FIGS. 5 a-5 d schematically illustrate video frame enhancement usingluminance and/or color data from a well-exposed image, and applying thedata to a relatively less exposed, sharper image in accordance with afourth embodiment which involves at least two image acquisitionsub-systems, so that the well exposed image is captured with a firstsub-system while the sharper image is captured with a second sub-system.

FIG. 6 schematically illustrates video and/or still frame enhancement inaccordance with certain embodiments.

FIGS. 7 a-7 d illustrate enhancement of a low-light image in accordancewith certain embodiments.

DETAILED DESCRIPTIONS OF THE EMBODIMENTS

A method of generating sharp, well-exposed, color images from low-lightimages is provided. A series of under-exposed images is acquired. A meanimage (Hmean) is computed based on the series of under-exposed images. Asum image (Gsum) is generated also based on the series of under-exposedimages. Chrominance variables of pixels of the mean image are mapped tochrominance variables of pixels of the sum image. Chrominance values ofpixels within the series of under-exposed images are replaced withchrominance values of the sum image. A set of sharp, well-exposed, colorimages is generated based on the series of under-exposed images withreplaced chrominance values.

The acquiring of the series of under-exposed images may includecapturing raw images and generating processed RGB images based on theraw images.

The computing of the mean image may include applying a tone-mappingtransformation to the series of under-exposed images. The tone-mappingtransformation may include a logarithmic transformation.

The mean and/or sum images may be transformed to YUV space prior tomapping and replacing the chrominance values. The under-exposed imagesmay be transformed with replaced chrominance values back to RGB space.

The set of sharp, well-exposed, color images may include multiple videoimages and/or one or more still images.

An image processing method in accordance with certain furtherembodiments is also provided. A first underexposed and sharp image of ascene is obtained. A second image relatively well exposed and blurredcompared to said first image, nominally of the same scene, is alsoobtained. The first and second images may be derived from respectiveimage sources, or from a same source, and/or may be based on or derivedfrom same or different sets of images taken from a same sequence ofacquired RAW images. The second image is acquired at a differentresolution and a different exposure time than the first image. Themethod involves scanning across the first and second images. A portionof the first image is provided as an input signal to an adaptive filter.A corresponding portion of the second image is provided as a desiredsignal to the adaptive filter. Filter coefficients are calculated forthe portion of the images at a given scanning location based on acombination of the input signal, the desired signal and existing filtercoefficients for the scanning location. The method further includesadaptively filtering the input signal based on the filter coefficientsto produce an output signal and constructing a first filtered image fromthe output signal, less blurred than the second image.

The first and second images may be in RGB format. The image portions mayinclude a respective color plane of the first and second images. Themethod may include constructing each color plane of the first filteredimage from a combination of the filter coefficients and the input signalcolor plane information.

The first and second images may be in YCC format. The image portions mayinclude a respective Y plane of the first and second images. Theconstructing of the first filtered image may include using the outputsignal as a Y plane of the first filtered image and using Cb and Crplanes of the input image as the Cb and Cr planes of the first filteredimage. The method may further include providing a portion of said firstfiltered image as the input signal to an adaptive filter; providing acorresponding portion of the second image as a desired signal to theadaptive filter; further adaptively filtering the input signal toproduce a further output signal; and constructing a further filteredimage from the further output signal less blurred than the firstfiltered image.

The first and second images may be in RGB format and, for producing thefirst filtered image, the image portions may include a respective colorplane of the first and second images. The providing a portion of thefirst filtered image may involve converting the first filtered image toYCC format. The method may include converting the second image to YCCformat. The image portions for further adaptive filtering may include arespective Y plane of the converted images.

The first and second images may be in YCC format and, for producing thefirst filtered image, the image portions may include a respective Yplane of the first and second images. The providing a portion of thefirst filtered image may involve converting the first filtered image toRGB format. The method may include converting the second image to RGBformat. The image portions for further adaptive filtering may include arespective color plane of the converted images.

The image source for the second image may be of a relatively higherresolution than the image source for the first image. The method mayinclude, prior to adaptive filtering, estimating a point spreadfunction, PSF, for the second image. Responsive to the PSF being lessthan a pre-determined threshold, the method may also include de-blurringthe second image using the point spread function.

The image source for the second image may be of a relatively lowerresolution than the image source for the first image. The method mayinclude aligning and interpolating the second source to match thealignment and resolution of the first source. Responsive to the firstand second sources being misaligned by more than a predeterminedthreshold, the method may include providing the desired signal from alinear combination of the first and second image sources, or providingthe desired signal from a combination of phase values from one of thefirst and second image sources and amplitude values for the other of thefirst and second image sources.

The adaptive filtering may be performed row-wise or column-wise on theinput signal. Further adaptive filtering may be performed the other ofrow-wise or column wise on the input signal.

The method may include amplifying the luminance characteristics of theunder exposed image prior to the adaptive filtering.

The first image may be one of an image acquired soon before or after thesecond image.

A digital video acquisition system is also provided, including a lens,an image sensor for capturing a stream of multiple digital, video orstill, images, a processor and a memory having processor-readable codeembedded therein for programming the processor to perform any of themethods described above or below herein.

One or more processor-readable media is/are also provided which haveprocessor-readable code embedded therein for programming one or moreprocessors to perform any of the methods described above or belowherein.

Several embodiments are described herein that use a pair of images toenhance one of them. Alternative embodiments may be obtained bycombining features of two or more of these embodiments, or by combiningfeatures of one or more of the embodiments with features described inthe background or in any of the references cited there or hereinbelow.In certain embodiments, one of the pair of images is the target image(which is sharp but under exposed and noisy), and another image is thereference image (which is well-exposed but blurred, e.g., motionblurred). In specific embodiments, the use of the method is dedicated toimproving the quality of video-frames, or sequences of still images,acquired in scenes with low light.

For ensuring animation (motion, fluency), the frame-rate in a video isgenerally over a predetermined threshold (e.g., 12 frames/sec). Thisimposes a maximum exposure time for each frame. In low light conditions,this exposure time is not-sufficient for offering good visual quality.Several embodiments are provided herein as solutions to overcome thisproblem. Several methods of improving video capture in low-light areprovided by applying techniques to a continuous sequence of images.Three embodiments generally incorporate modified image processing withina conventional video camera or any later generation camera, while afourth embodiment involves a video camera that incorporates two distinctimage acquisition subsystems.

Image+Video

FIG. 1 schematically illustrates a conventional frame acquisitionprocess. According to the illustration of FIG. 1, multiple video frames110 are sequentially acquired each with a same exposure duration.

FIG. 2 schematically illustrates use of frames with different exposuredurations. A well-exposed frame 210 is acquired at a beginning of avideo acquisition. From this well-exposed image 210, one may extractbrightness and/or color information that are used to enhance the nextframe 220. The next frame 220 is exposed using an exposure determinedfrom a maximum frame-rate. The frame 220 is enhanced with brightnessand/or color information extracted from the well exposed frame 210.

The frame 220 may be displayed (included in the video) and may be usedfor determining or generating another reference image 230 that mayitself be used to enhance the next acquired frame 240 (frame 2). Thereference frame 230 may be generated as a combination of the firstacquired frame 210 and the second acquired frame 220.

The next acquired frame 240 is enhanced with brightness and/or colorinformation extracted from the generated well exposed frame 230. Theframe 240 may be displayed (included in the video) and may be used fordetermining or generating another reference image 250 that may itself beused to enhance the next acquired frame 260. The reference frame 250 maybe generated as a combination of the generated frame 230 and the thirdacquired frame 240. The frame 260 may be displayed (included in thevideo) and the process may be repeated over a complete vide sequence.

The acquisition of the well exposed image may be repeated at certaintimes. A digital camera may be used that has an auto-exposure algorithmthat runs in the video mode. The best exposure time for a frame may beestimated in-camera. The camera may be configured such that a wellexposed image may be acquired only when the auto-exposure demands aperiod larger than the one imposed by the maximum frame rate.

If the entire video is acquired in low light then the amount of timewhen it is needed to repeat the image acquisition may depend on cameramovement. This may be estimated by registration techniques. An examplemethod may be performed by a processor as follows:

-   -   1. Test for Low-Light Case: the auto-exposure determined time is        approximately equal to the maximum frame rate. The test should        run continuously no matter which is the current case.    -   2. When the conditions are such that it is not a Low-Light Case,        then normal acquisition may be performed with record of any well        exposed image.    -   3. When the conditions are such that it is a Low-Light Case,        then:        -   a. Acquire a well exposed image and iteratively run the            enhancing method for each video frame;        -   b. Estimate the camera movement by performing registration            on the recorded frames. If the motion is larger than a            threshold (the image content has significantly changed,            e.g., 30% from the last well exposed image), then acquire a            new well-exposed image; and        -   c. If a certain number of video frames (e.g., M=5, 10, 20,            30, 40, 50, 60, 75, or 100 or more) has been recorded from            the last well-exposed image, then acquire a new well exposed            image.

Video Frame Pairs with Modified Exposure Time

FIG. 3 schematically illustrates use of video frame pairs with modifiedexposure times. In an example embodiment, we will consider only a video.The exposure times of the frames are modified from the example of FIG. 1to provide a better response to a low-light scene. Traditionally, theexposure times of all the frames are equal (with t₀) as illustrated atFIG. 1. In the example of FIG. 3, the images may be grouped in pairs210, 220, wherein the exposure time of a pair is 2t₀, just as are twoframes 110 in FIG. 1. FIG. 3 differs from FIG. 1, however, because for apair in FIG. 3, there is an under-exposed frame 310 (with exposure timekt₀, k<1) and an “over-exposed” image 320 (with exposure time of(2−k)t₀). The “over-exposed” frames 320 are used for a better detectionof the appropriate luminance level and of the color gamut, and thereforeto enhance the under-exposed frames 310.

Choosing k: The enhancement method has a limitation. There is a maximumdifference between the exposures of the two images (well exposed 320 andunder-exposed frame 310), when the method works optimally. If theexposure difference is larger than this limit, the under-exposed image310 will have more noise than is typically tolerated in a useful signal.The k is preferably chosen as small as possible (to expose as well aspossible the “well-exposed image” 320), but its value is preferably keptin the mentioned method limits. Also, the exposure time of thewell-exposed image 320, (2−k)t₀, is preferably not larger than the valuechosen by the auto-exposure (which may be used as a global reference).

Video Frame Triplets with a Single Well Exposed Image

FIG. 4 schematically illustrates a further embodiment which uses videoframe triplets including one (or more) well-exposed image(s) 420 eachwith two or more under-exposed frames 410, 412. This embodiment providesa natural extension of the previous one described above with referenceto FIG. 3, because the enhancement method illustrated at FIG. 3 does notrequire an order of the two images 310, 320. Therefore, the luminanceand/or color information from the well exposed image 420 may be used toenhance the previous frame 410, as well as in the next frame 412. Insuch a scheme, as illustrated at FIG. 4, the maximum exposure may reachhigher values, and the overall frame rate may be better from the pointof view of the edges. In principle, each well-exposed image 420 may beused to enhance still further combinations of under-exposed images,e.g., two before and/or two after, or one before and two after, or oneafter and two before, or three before and/or after, etc.

Dual Frame Acquisition Pipeline

FIGS. 5 a-5 d schematically illustrate dual frame acquisition pipelineembodiments. FIGS. 5 a-5 d schematically illustrate video frameenhancement involving use of luminance and/or color data from awell-exposed image, and applying the data to one or more relatively lessexposed, sharper image(s). FIGS. 5 a-5 d illustrate four alternativeexamples that are each in accordance with a fourth embodiment whichinvolves at least two image acquisition sub-systems, where the timeparameter is understood to go from left to right in FIGS. 5 a-5 d fromearlier to later. In general, a well exposed image is captured with afirst image acquisition sub-system that is optimized for longerexposure, while a relatively less exposed and sharper image is capturedwith a second sub-system that is particularly optimized for sharpness.

FIG. 5 a illustrates short-exposure time (SET) frames 502 which aresharp, but underexposed. These frames 502 are aligned with the start ofthe full exposure time (FET) frames 504 with normal exposure, fullcolor, but blurred. These frames 502, 504 are combined to generateenhanced video frames 508. FIG. 5 b shows the case where the SET frames502 are acquired towards the end of the FET frames 504. This illustratesthat SET frames 502 can be acquired any time during the FET frame 504 inthis embodiment.

FIGS. 5 a-5 b illustrates an embodiment wherein the elements 502indicate video frames 502 that are acquired with shorter exposure timesthan element 504. This can be understood from the narrower extent of theframes 502 in the time dimension than the frames 504. The frames 504 areexposed for two to ten times as long (or longer, or just under 2× insome embodiments) compared with the frames 502, while in FIGS. 5 a-5 dthe frames 504 are about 5× wider than the frames 502 indicating anexposure time of about 5× longer than for frames 504. The imagecombining unit (ICU) 506 combines luminance and/or color informationfrom the frames 504 to enhance the frames 502. The combinations areimages 508.

In FIG. 5 a, each image 502 is acquired beginning about the same time asa corresponding image 504, even though the exposure of image 502 isended longer before the exposure of image 504 is ended. In FIG. 5 b,each image 502 is acquired in the last moment of acquisition of acorresponding frame 504, i.e., the exposures of the two IRS's are endedat about the same time even though the exposure of frame 504 was begunfar sooner than that of image 502. Another embodiment would have theexposure of image 504 begin before the beginning of the exposure ofimage 502, and the exposure of image 502 is ended before the exposure ofimage 504 is ended, i.e., the frame 502 is acquired in the somewhere inthe middle of the exposure of frame 504.

FIG. 5 c illustrates a case of multiple SET frames 512 acquired withinan exposure duration of a single FET frame 514. Each SET frame 512 canact as a “sharpening reference” for the FET frame 514 enabling multipleenhanced video frames to be resolved. This is advantageous for examplein very low light, where it might take as long as 0.5 seconds for asensor to achieve a full exposure. Using multiple SETs 512 within that0.5 seconds permits a reasonable frame rate of 10-15 fps to be achievedwith enhanced output frames in terms of sharpness, color and/orluminance.

FIG. 5 c schematically illustrates a further embodiment where multiple,e.g., three, short exposure time (SET) frames 512 are acquired while asingle longer exposure time frame (FET) 514 is acquired. In anotherexample, five short exposure time frames 512 are acquired while a singlelonger exposure time frame 524 is acquired. In another example, sevenshort exposure time frames 512 are acquired while a single longerexposure time frame 534 is acquired. Alternatively, the shorter exposuretime frames could be varied as well. In the first example of three SETsper FET, the ICU 516 combines the color and/or luminance informationfrom frame 514 with each of the three frames 512 that were acquiredduring the exposure duration of the frame 514. The results are the threeoutput images 518.

In FIG. 5 c, the second (and subsequent) SET frames 512 could becombined with the first output video frame 518 rather than with the mainFET frame 514 in the same way as an embodiment of the single acquisitionchannel aspect. Thus, in the example of FIG. 5 d, the SETs and FETs maybe similar to those described with reference to FIG. 5 c. In thisexample, the color and/or luminance information initially extracted fromframe 524 is applied to 522 at ICU 526 to get a first processed frame528. In this example, color and/or luminance information is extractedfrom that first processed frame 528 and combined at ICU 526 with one ormore subsequent shorter exposure frames 522 to get one or moresubsequent processed frames 528. This example is particularly applicablealso to the single IAS embodiment except without having temporaloverlap.

In this fourth embodiment, a video camera incorporates two distinctimage acquisition subsystems (IAS). In certain embodiments, each of thetwo IRS's includes at least an image sensor & (digital) post-processingcomponents.

The first IAS may have an image sensor configured to acquire images at ahigher frame rate and with optimal image sharpness. The frame rate forthis first sensor may optionally be fixed and it may optionallyincorporate an image buffer to temporarily store a plurality of thesesharp images.

The second image acquisition subsystem may be configured to capture anoptimally exposed image. As such, it should have a variable frame rateto allow for correct image exposure, even in low lighting conditions.

A timing or clocking unit may be used to maintain temporalsynchronization between the two IRS's and to ensure that images in thefast IAS image buffer can be correctly matched to images obtained fromthe optimally exposed IAS. This timing function may be implementedeither as a hardware unit, or as part of the firmware of a main CPU orDSP.

An additional image combining unit (ICU) selects one (or more) imagesfrom the fast IAS buffer and a corresponding image from the optimallyexposed IAS buffer. The ICU may perform various operations such as thosedescribed at U.S. Ser. Nos. 12/330,719 and/or 11/856,721, which areincorporated by reference. The ICU may also be used with any of thefirst, second or third embodiments described above with references toFIGS. 2-4. The ICU generates a sequence of output images which combinethe one or more sharper, underexposed images from the fast IAS bufferwith the single optimally exposed image obtained from the optimallyexposed IAS.

The system of the fourth embodiment may acquire images of both the fastIAS and the optimally exposed IAS with a same lens as optical input.That input is then split first and second image sensors. Alternatively,images may be acquired separate lens/sensor combinations. Such multiplelens subsystem may include 2, 3, 4 or more lenses. For example, U.S.Pat. No. 7,453,510, which is incorporated by reference, describes amulti-lens subsystem having particular embodiments with four lenses.

Parallax correction may also be provided, as has been provided in thepast in “twin reflex” cameras. When the imaging device incorporates asensor for measuring distance-to-subject, the resulting measure to theprinciple subject of the image may be employed to further enhance theparallax correction, and/or to determine if such correction is needed.When the imaging device incorporates a face detector and/or a facetracking unit, information regarding the size(s) of face(s) detected inthe image may be used to further enhance the parallax correction, and/orto determine if such correction is needed. In general, when a subject isbeyond a certain distance or “critical distance” (infinite focallength), e.g., three or four meters in common cameras, then parallaxcorrection loses its advantage.

Techniques described in US published application 2007/0296833 and/orU.S. Ser. No. 12/116,140, which are incorporated by reference, may becombined with features described herein, particularly to improve theacquisition, or pre-processing of the input images to the ICU, and/or toselectively enhance the output images from this unit.

Output images may be subsequently compressed to form an MPEG videosequence. US published application 2007/0025714 is incorporated byreference. In addition, the techniques described herein may be appliedto a still image camera, whereas the sequence of images may be preview,post-view and/or in-whole or in-part contemporaneously acquired imagesthat are generally captured to enhance capture conditions and/orpost-processing of a main still image.

In a variant of the dual IAS embodiment, both IAS subsystems may share acommon image sensor which employs a modified architecture. Thisembodiment does not have separate optical acquisition subsystems andassociated parallax correction which would be involved in capturingclose-up or portrait images.

In this variant embodiment, an image sensor may be partitioned such thatevery second row of sensor pixels is tied to a separate clock circuit.Thus, the acquisition time of the odd-numbered rows of the image sensorcan be controlled independently of the even-numbered rows. Similarly thedata outputs (pixel values) of these even and odd rows can be loadedonto independent data buses. Thus, by setting different acquisitiontimes for the odd and even rows of sensor pixels, closely overlappingunderexposed and normally exposed images may be advantageously acquired.Due to the close proximity of the odd and even pixel rows, there is asmall, regular half-pixel offset in either horizontal or verticalorientation which can be compensated for, and as both underexposed andnormally exposed images are acquired via that same optics subsystem,there is no parallax correction involved. As in other embodiments, thetwo independently acquired images may be combined using a single passadaptive filter.

Bayer Pattern Partitioning of the Sensor

In another embodiment pairs of rows are linked for each of the differentexposure times. Thus, an entire RGB Bayer set corresponding to a singleRGB pixel will have the same exposure time i.e., rows n+1, n+2 have ashort exposure, while rows n+3, n+4 have a normal (less sharp orrelatively more blurred) exposure.

Considering the exposure time of the underexposed rows is T_(U), whilefor the normally exposed rows is T_(N), an example acquisition procedureis as follows:

-   -   Start exposure for all rows of the image sensor (i.e., for both        the underexposed rows and the normally exposed rows);    -   After (T_(N)−T_(U)) seconds, reset (or discharge) the odd rows;    -   After T_(N), all the rows are read (i.e., both the underexposed        and normally exposed rows). Two images are formed with half        resolution; and    -   The underexposed image is interpolated to full resolution.

An example choice is to have T_(N)=k T_(U), where k is an integerscalar. This means that the normal exposure time is a multiple of theshort exposure time. Such constraint may be met, e.g., if the exposuretime is reduced by an amount of Ev (or Fstop) values. For instance, anunder-exposure with one (1) Fstop implies T_(N)=2 T_(U). This constrainthas at least the following benefits:

-   -   Using the same sampling frequency, all the locations (from the        image sensor) can be read synchronously;    -   When all the operations are pixel-wise (or use only pixels from        two consecutive rows), there is no need for an extra image        buffer. The resulting image is built in real-time, while        reading. The interpolation operation is performed locally.

An advantage of this split-sensor acquisition embodiment is that the twoimages are spatially aligned (insofar as is possible given theconstraint that one image is blurred) and hence pixel-wise (for eachlocation independently) operations are possible without a need toacquire and align the entire images.

A further advantage is that the acquisition of the two images canhandled by two independent data busses, enabling the two images to beobtained with a close temporal overlap.

Image Mixing

This offers an alternative approach to the combination of the two imagesusing a single pass adaptive filter. In one example, a process mayinvolve the following two steps:

The first step is about combining the underexposed image F and theblurred image G, by some point-wise amplification algorithm to producean image, called F_(amp), and which has a luminance and color levels atleast very close to the ones present in the blurred image G. Theresulting image F_(amp) has reasonable, but not perfect, luminance andcolor levels. Because the amplification is performed on each pixelindependently, the amplified image will have the same resolution as theunder-exposed image, which is the desired resolution.

As a second step, the final image, F_(out) may be computed as a linearcombination (implemented independently on each pixel) between theblurred image, G, and the amplified image F₁. An example solution toperform this operation is:

-   -   Use the YUV plane for performing the calculus;    -   A luminance of the output image may be expressed as:        Y(F _(out))=0.9*Y(F _(amp))+0.1*Y(G);    -   A color difference plane of the output image may be expressed        as:        U(F _(out))=0.3*U(F _(amp))+0.7*U(G),  (i)        V(F _(out))=0.3*V(F _(amp))+0.7*V(G).  (ii)

The system of the fourth embodiment may be applied to other fields ofimage processing. For example, improved Dynamic Range Compression (DRC)may be achieved using a pair of symmetrical, synchronized sensors. Inthis embodiment, one of the pair of symmetrical, synchronized sensorsmay be used to expose correctly, while the other is used to captureover-exposed images, or under-exposed images. The dark areas, or lightareas, can be improved drastically, e.g., in terms of increasedsharpness, color balance, and/or white balance, and/or reducedblurriness.

In another example, hand motion compensation and low lightingvideo/still image enhancement can be improved. In this example, onenormally exposed image and another under-exposed image (e.g.,half-exposed, one-third exposed, one-quarter exposed, one-fifth exposed,or less). Adaptive filtering may be used to combine the images.

In a further embodiment, a third dimension may be added to an image.This feature can be used in an enhanced face recognition technique, orto render, process, store, transmit or display an improved 3Drepresentation.

In another embodiment, improved face tracking is performed. In thiscase, two face detections are performed on two different image streams.One detection may be over-exposed (or under-exposed), and the other maybe normally exposed. In this case, e.g., back-lit objects and otherdifficult lighting conditions can be advantageously handled moreefficiently and effectively and with improved results.

In another embodiment, red eye detection and correction is improvedusing a still image version of the dual frame pipeline embodiment. Therecan appear color exact shifts between pictures obtained by two suchsensors. That can provide an advantageous verification filter, e.g., toconfirm that detected red pixels are indeed red eye artifacts.

In a further embodiment, foreground/background separation techniques canbe enhanced using a dual frame pipeline. For example, one sensor maycapture a focused image, while the other is used to capture an at leastslightly de-focused image. Advantageous separation between backgroundand foreground can be thereby achieved allowing for the creation of highquality portraits with inexpensive and/or relatively simple optics.

In another example, one sensor may be used to capture IR images orimages that include an IR component. Such sensor can be provided byremoving an IR filter of certain sensors and/or adding a visible lightfilter. Such IR sensor would be advantageous in performing imageprocessing and image capture in lowlight conditions, and in a stillimage embodiment for performing red eye detection and correction.

Other examples include noise reduction and/or processes involvingcapturing non-scene shot images to be further used as additional camerafeatures. For example, an eye iris analysis tool may be embedded in adigital cameras that has an eyepiece viewfinder. The image of the irislooking through the eyepiece may be captured by the camera CCD, or anadditional CCD, after being diverted by one or more optics. The image ofthe iris may be analyzed and used for biometric user identification(camera lock, digital picture watermarking), inferring lightingconditions, detecting user fatigue, and/or detecting certain symptoms ofa medical condition of a user.

A low-resolution camera looking back at a user's face may be embedded ina digital camera. The image of the user's face may be captured by thecamera CCD, or an additional CCD, after being acquired and processed byone or more optics. This can be used for face identification for camerausage protection, IP protection for digital pictures, and/or analysis ofuser emotions based on the face and/or as feed-back of the user emotionsinto the captured image itself.

In another embodiment that is particularly advantageous for lowlightconditions, a reduced frame rate may sometimes be used in order toincrease the exposure duration to gather more light. If multiple sensorsare used, e.g., two (although three or more may be used), then a fasterframe rate will result in sharper images. For example, even frames maybe acquired with one sensor, and odd frames with another sensor (or forthree sensors, every third frame is acquired with each of the threesensors, or for four sensors, etc).

The sensors will generally have the same exposure (e.g., for twosensors, the time T may be about twice the frame rate), but one sensorwill start acquiring image data at t=0, and the second sensor will startat t=T/2.

Alternative to the previous embodiment that is particularly advantageousfor lowlight conditions, instead of or in addition to increasing (ordecreasing) exposure time, a larger aperture can be set. While depth offield will be smaller such that a subject with background would bedefocused, the frame rate can be advantageously higher than the exampleof increasing the exposure time. With dual sensors and optical elements,both lenses can be set to large aperture, where one is focused on thebackground and the other is on the subject. The two resulting picturescan be combined, in one high-frequency picture, where both the subjectand background are in focus (the two images are complementary, where onehas high frequencies the other has low frequencies). So in lowlight,exposure time can be kept short and ISO small, with fully open aperture,wherein two short DoF images are combined into one with good focus bothon subject and background.

In a de-noising algorithm, two images can be acquired with oneoverexposed and the other normally exposed. The normally exposed imagemay be de-noised using information from the overexposed image. Anadaptive algorithm may be used as a function of illumination of thepixels, because lower illuminated pixels exhibit greater noise.

In a further embodiment, a two lens system can be used in anotherembodiment as follows. One lens may capture certain scenery broadly,e.g., a tourist either standing relatively close to a camera with theEgyptian pyramids far away from the camera and tourist, or the touriststanding further from the camera and closer to the pyramids and lookingvery small. The other lens is used to zoom and focus on the tourist orother main subject from the scene.

A single file or multiple files may then store the two images. The twoimages can be acquired at the same time with a system in accordance withthe dual image pipeline embodiment, or proximate in time in the singleoptical system embodiments. An advantage is that, while having apanoramic image, there will still be excellent detail in the main partor subject of the scene (e.g., the tourist or more particularly the faceof the tourist). In one embodiment, the wide-angle scene may use a facetracker to determine the location of a face or faces in the scene, andthe zoom/aperture of the second imaging system may be adjusted toinclude people detected in the captured image. In an alternativeembodiment, both wide-angle and narrow angle images of the scene may becaptured and subsequently combined into one or more composite images. Inanother embodiment, selective high-quality compression of a face regionmay be enabled in a portrait (e.g., using a zoom lens), withlower-quality compression applied to the background image (e.g., using awide-angle lens).

Wavefront coding may also be provided. Variable focus is used fordifferent parts of a lens. Various parts of the lens capture informationrelating to different focal points. The image is decoded according toknown defocus variations and a resulting image has an extended depth offield. CDM Optics have studied this technology, andhttp://www.cdm-optics.com is incorporated by reference.

Phase retrieval and/or phase diversity may also be provided. Two imagesare captured simultaneously with a system in accordance with the dualpipeline embodiment. One image is in focus, and another is defocused bya known distance. The two images are recombined and the known defocusdistance which distorts the image is used to reconstruct the image witha greater depth of field. A beam splitter could also be used to capturethe defocused image to apply phase diversity.

Tone Mapping for Low Light Image Enhancement

A tone mapping function may be applied by differentiating two imageswhich have identical or approximately the same spatial content. Thedifference between the two may be the intensity of light and of thecolors. That is, one is well-exposed while the other is under-exposed.In accordance with certain embodiments, the matching may be generallyexact, without any approximation, due to the same raw images being usedto generate mean and sum images with which the tone mapping isperformed.

Movies taken in low-light conditions, at 30 fps, tend to result in verydark, under-exposed images that have a very low contrast. Improving suchimages implies using tone mapping algorithms that, although give abetter exposed image, have many difficulties in restoring the naturalcolors of the scene. On the other hand, images taken in the sameconditions at a sufficient exposure time have natural colors and a goodcontrast but a poor quality when local or global motion is present. Atone mapping function may be advantageously determined in accordancewith certain embodiments by differentiating two images which haveidentical or substantially the same spatial content.

An advantageous technique works in accordance with certain embodimentsas follows. For a sequence of under-exposed images, the equivalentblurred but well-exposed image is computed from the sum of the images inRAW format (Fsum). Fsum=F1+F2+ . . . +FN, where: F1, F2, . . . , FN, arelow light RAW images—The same set of images F1, F2, . . . , FN isavailable as final (camera processed) images G1, G2, . . . , GN. This istrue for the sum image: Fsum->Gsum. The Gi images are given in the RGBcolor space.—

After the in camera processing is done, a mean image for the sequence offinal images is computed: Gmean=(G1+G2+ . . . +GN)/N. The Gmean imagehas similar exposure and contrast, as well as colors, as the images inthe sequence (G1, G2, . . . , GN), but has the exactly same content asthe well-exposed image (Fsum). Both are degraded by the same motionblur.—The underexposed images, G1, G2, . . . , GN, are improved by alogarithmic tone mapping transformation which increases contrast andvisibility but does not give the same colors as a good exposure. Thetone mapping transformation may be represented as follows: Hi=f(Gi),where f is a logarithmic tone mapping function.

A RGB->YUV transformation is applied. Hence Hi_YUV are given in the YUVcolor space.—In the same manner, Gsum_YUV is obtained out of Gsum (whichcame from the sum of the RAW images, Fsum)—Now the Gsum_YUV(well-exposed image) are compared with Hmean_YUV (under-exposedimage).—Working on the YUV color space, the chrominance pairs from thetransformed under-exposed image (Hmean_YUV) are mapped to chrominancepairs (U,V) computed from the well-exposed image (Gsum_YUV).

The pixels that have a certain chrominance pair in (Hmean_YUV) will bemapped to a chrominance pair computed as the mean of the chrominancepairs of the corresponding pixels in Gsum_YUV.—For each image, Hi_YUV,in the sequence, the chrominance pairs of the pixels are replaced by thecorresponding chrominance pair from the maps previously created. Theimages can have slightly different chrominance which are not found inthe mean image, in which case these may be replaced by the closest pairof chrominance (using Euclidian distance) found in the mean image, whilethe differences are unnoticeable to the human eye.—The YUV->RGBtransformation is applied and the transformed image has similar colorwith the well-exposed image, without being affected visibly by themotion blur.

The process may be implemented on a typical digital camera platform thatprovides special access to the RAW images. The computation can be donewith a typical CPU or with dedicated ASIC.

FIG. 6 schematically illustrates video and/or still frame enhancement inaccordance with certain embodiments. The n frames of RAW image data 602are summed 604 to obtain a RAW sum image 606. A JPG sum image 608 basedon the RAW sum image 606 is also generated. This JPG sum image 608contains sufficient chrominance information, whereas the n frames of RAWimage data 602 are based on “under-exposed” images which haveinsufficient chrominance information. The under-exposed RAW images arehowever sharper than the sum image 608. Put differently, the sum imagetends to have sufficient chrominance information but is blurry, similarto a well-exposed or otherwise sufficiently exposed image (i.e., havingsignificantly longer exposure than that of the RAW images 602),particularly in comparison to the sharp, under-exposed images. Inaddition, frame by frame JPG images 610 are generated based on the nframes of RAW image data 602. A mean 612 is computed of the JPG images610 as a mean JPG image 614. Chrominance variables of pixels of the meanimage 614 are mapped to chrominance variables of pixels of the sum image608, and chrominance maps 616 are generated. The frame by frame JPGimages are chrominance mapped 618, and n outframes 620 are obtained as aresult.

This advantageous solution illustrated schematically in FIG. 6determines a tone mapping function by differentiating two images whichhave substantially identical spatial content; the difference between thetwo is the intensity of light and color: one is well-exposed while theother is underexposed. Because the sum image 608 is based on a sum ofthe RAW image 602, compared to being separately acquired from the RWimages 602, the matching is more exact, such that approximations foralignment and pixel-wise mapping are less dramatic, reduced oreliminated altogether.

An advantageous algorithm in accordance with certain embodiments cancombine information from both of the previously mentioned situations.The following is a non-limiting example. For a sequence of under-exposedimages, the equivalent blurred but well-exposed image is computed fromthe sum of the images in RAW format (Fsum). For example, Fsum=F1+F2+ . .. +FN, where F1, F2, . . . , FN, are low light or under-exposed RAWimages.

The same set of images F1, F2, . . . , FN is available as final (cameraprocessed) images G1, G2, . . . , GN. This is true for the sum image:Fsum->Gsum. The Gi images may be given in the RGB color space in thisexample.

After all in camera processing is done, a mean image for the sequence offinal images is computed: Gmean=(G1+G2+ . . . +GN)/N. The Gmean imagehas similar exposure and contrast, as well as colors, as the images inthe sequence (G1, G2, . . . , GN), but has the same content as thewell-exposed image (Fsum). Both are degraded by the same motion blur.

The underexposed images, G1, G2, . . . , GN, are improved by alogarithmic tone mapping transformation in accordance with certainembodiments, which increases contrast and visibility but does not givethe same colors as a well-exposed, or sufficient exposed image. The tonemapping transformation may be characterized as Hi=f(Gi), where f may bea logarithmic tone mapping function, or an alternative tone or colormapping function.

A RGB->YUV transformation may be next applied. Hence Hi-YUV may be givenin the YUV color space. In similar manner, Gsum-YUV may be obtained fromGsum (which came from the sum of the RAW images, Fsum).

Now the Gsum-YUV (well-exposed image) is compared with Hmean-YUV(under-exposed image). Working within the YUV color space, thechrominance pairs from the transformed under-exposed image (Hmean-YUV)are mapped to chrominance pairs (U,V) computed from the well-exposedimage (Gsum-YUV). The pixels that have a certain chrominance pair in(Hmean-YUV) will be mapped to a chrominance pair computed as the mean ofthe chrominance pairs of the corresponding pixels in Gsum YUV.

For each image Hi-YUV in the sequence, the chrominance pairs of thepixels are replaced by the corresponding chrominance pairs from the mapspreviously created. The images can have slightly different chrominancewhich are not found in the mean image, in which case these may bereplaced by the closest pair of chrominance (using Euclidian distance)found in the mean image. These differences are generally unnoticeable tothe human eye.

The YUV->RGB transformation is now applied. The transformed image hassimilar color with the well exposed image, yet advantageously withoutbeing affected visibly by the motion blur.

FIGS. 7 a-7 d illustrate enhancement of a low-light image in accordancewith certain embodiments. FIG. 7 a illustrates a mean image of thesequence of low-light (under-exposed) images. An initial, under-exposedimage is illustrated at FIG. 7 b. As shown, the under-exposed(low-light) image of FIG. 7 b is sharp yet fails to have optimum colorand brightness. Corresponding correctly exposed images are illustratedin FIG. 7 c showing sufficiently color and brightness, yet being blurry.An example of a final, enhanced image is shown in FIG. 7 d. Clearly, thefinal image of FIG. 7 d is sharper than the well-exposed or otherwisesufficiently exposed image of FIG. 7 c, and has improved color andbrightness compared with the under-exposed images of FIGS. 7 a and 7 b.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention as set forth in the claims that follow and their structuraland functional equivalents. For example, any of the followingalternative features may be combined into alternative embodiments:

Alternative Embodiments

A method of combining image data from multiple frames to enhance one ormore parameters of digital image quality is provided, which uses aprocessor and at least one video acquisition system including a lens andan image sensor. The method includes acquiring a first image at a firstexposure duration, as well as acquiring a second image at a secondexposure duration shorter than the first exposure duration and at a timejust before, just after or overlapping in time with acquiring the firstimage, such that the first and second images include approximately asame first scene. In this way, the second image is relatively sharp andunder-exposed, while the first image is relatively well-exposed and lesssharp than the second image. Brightness and/or color information areextracted from the first image and applied to the second image. Thebrightness and/or color information is/are extracted from the firstimage to generate an enhanced version of the second image enhanced interms of brightness and/or color quality. The enhanced version of thesecond image is displayed, stored, transmitted, and/or streamed. Thedigital images may be video or still images.

The method may further include applying to one or more further imagesthe brightness and/or color information extracted from the first imageto generate enhanced versions of the one or more further images enhancedin terms of brightness and/or color quality. The one or more furtherimages may be acquired at or near the second exposure duration and at atime or times just before or just after, or both, the acquiring of thesecond image, such that the one or more further images includeapproximately the same scene as the first and second images.

The method may also include acquiring a third image at or near the firstexposure duration when a predetermined number of multiple frames hasbeen recorded since the acquiring of the first image. A fourth image mayalso be acquired at or near the second exposure duration and at a timejust before, just after or overlapping in time with the third image,such that the third and fourth images include approximately a samesecond scene different from or same as the first scene. Brightnessand/or color information are extracted from the third image, and appliedto the fourth image to generate an enhanced version of the fourth imageenhanced in terms of brightness and/or color quality. The enhancedversion of the fourth image is displayed, stored, transmitted, and/orstreamed.

The method may further include acquiring a third image at or near thefirst exposure duration when a predetermined amount of camera movementhas been detected since the acquiring of the first image. A fourth imagemay be acquired at or near the second exposure duration and at a timejust before, just after or overlapping in time with the third image,such that the third and fourth images include approximately a samesecond scene different from the first scene due at least to the cameramovement. Brightness and/or color are extracted from the third image,and applied to the fourth image to generate an enhanced version of thefourth image enhanced in terms of brightness and/or color quality. Theenhanced version of the fourth image is displayed, stored, transmitted,and/or streamed.

The method may also include acquiring a third image at or near the firstexposure duration including combining some of the data from the firstimage with some or all of the data from the second image. A fourth imagemay be acquired at or near the second exposure duration and at a timejust before, just after or overlapping in time with the third image,such that the third and fourth images include approximately the samefirst scene. Brightness and/or color information are extracted from thethird image and applied to the fourth image to generate an enhancedversion of the fourth image enhanced in terms of brightness and/or colorquality. The enhanced version of the fourth image is displayed, stored,transmitted, and/or streamed within a video or still image sequence. Themethod may include repeatedly iterating the method as the scene evolveswith time and/or camera movement.

The method may include iteratively alternating acquisition ofwell-exposed and sharp, under-exposed images; extracting brightnessand/or color information from the well-exposed images; and applying tothe sharp, under-exposed images the brightness or color information, orboth, extracted from the well-exposed images to generate enhancedversions of the sharp, under-exposed images enhanced in terms ofbrightness or color quality, or both; and displaying, storing,transmitting, or streaming the enhanced versions of the sharp,under-exposed images.

The method may also include iteratively performing the following:

-   -   acquiring two sharp, under-exposed frames and a well-exposed        frame;    -   extracting brightness and/or color information from the        well-exposed frame;    -   applying to each pair of sharp, under exposed images        respectively captured immediately before and immediately after        the acquiring of each well-exposed frame the brightness and/or        color information extracted from the well-exposed frame to        generate enhanced versions of each of the pair of sharp,        under-exposed images enhanced in terms of brightness and/or        color quality; and    -   displaying, storing, transmitting, or streaming the enhanced        versions of the pair of sharp, under-exposed images.

The method may utilize separate image acquisition subsystems (IRS's) foracquiring the first and second images. The separate IRS's may includefirst and second IRS's, wherein the first IAS is configured to acquireimages at a faster frame rate than the second IAS. The first IAS may bespecially configured to capture relatively sharp, under-exposed imagesand the second IAS may be specially configured to capture well-exposedand less sharp images. The first IAS may have a fixed frame rate. Thesecond IAS may have a variable frame rate configured for capturingoptimally-exposed images under different lighting conditions. The methodmay include acquiring both the first and second images with a same lensas optical input, and splitting the optical input between first andsecond image sensors. The method may alternatively include acquiring thefirst and second images, respectively, with a first lens/sensorcombination and a second lens/sensor combination. In this alternative,parallax correction may be applied.

In accordance with another embodiment, an image processing apparatus isarranged to process a first underexposed and sharp image of a scene, anda second image relatively well exposed and blurred compared to saidfirst image, nominally of the same scene. The first and second imagesare derived from a same image source or from respective image sources inthis embodiment. An adaptive filter may be arranged to scan across thefirst and second images to provide a portion of the first image as aninput signal to the adaptive filter. A corresponding portion of thesecond image may be provided as a desired signal to the adaptive filter.Filter coefficients may be calculated for this portion of the imagesbased on a comparison of the input signal and a function of the desiredsignal and existing filter coefficients. The adaptive filter may bearranged to produce an output signal from the input signal based on thefilter coefficients. An image generator may be arranged to construct afirst filtered image from the output signal, relatively less blurredthan the second image.

The first and second images may be in RGB format and the image portionsmay include a respective color plane of the first and second images.Each color plane of the first filtered image may be constructed from acombination of the filter coefficients and the input signal color planeinformation.

The first and second images may be in YCC format and wherein the imageportions may include a respective Y plane of the first and secondimages. The constructing of the first filtered image may involve use ofthe output signal as a Y plane of the first filtered image and Cb and Crplanes of the input image may be used as the Cb and Cr planes of thefirst filtered image.

A portion of the first filtered image may be provided as the inputsignal to an adaptive filter. A corresponding portion of the secondimage may be provided as a desired signal to the adaptive filter. Thetechnique may further include adaptively filtering the input signal toproduce a further output signal. A further filtered image may beconstructed from the further output signal less blurred than the firstfiltered image.

The first and second images may be in RGB format and, for producing thefirst filtered image, the image portions may include a respective colorplane of the first and second images. The providing of a portion of thefirst filtered image may include converting the first filtered image toYCC format. The second image may be converted to YCC format, the imageportions for further adaptive filtering may include a respective Y planeof the converted images.

The first and second images may be in YCC format and, for producing thefirst filtered image, the image portions may include a respective Yplane of the first and second images. The providing of a portion of thefirst filtered image may include converting the first filtered image toRGB format. The second image may be converted to RGB format, and theimage portions for further adaptive filtering may include a respectivecolor plane of the converted images.

The image source for the second image may be of a relatively higherresolution than the image source for the first image. The first imagesource may be aligned and interpolated to match the alignment andresolution of the second image source. A point spread function, PSF, maybe estimated for the second image. The second image may be de-blurredwith the point spread function. The de-blurring may be performed inresponse to the PSF being less than a pre-determined threshold.

The image source for the second image may be of a relatively lowerresolution than the image source for the first image. The second sourcemay be aligned and interpolated to match the alignment and resolution ofthe first source. It may be determined that the first and second sourcesare misaligned by more than a pre-determined threshold, and the desiredsignal may be provided from a linear combination of the first and secondimage sources. Responsive to determining that the first and secondsources are misaligned by more than a pre-determined threshold, thedesired signal may be provided from a combination of phase values fromone of the first and second image sources and amplitude values for theother of the first and second image sources.

The adaptive filtering may be performed row-wise and/or column-wise onthe input signal.

The luminance characteristics of the under exposed image may beamplified prior to the adaptive filtering.

Noise filtering may be performed on the first filtered image. Colorcorrection may be applied to the first filtered image and/or the noisefiltered image.

The first image may be one of an image acquired soon before or after thesecond image, or temporally overlapping the second image.

A technique in accordance with certain embodiments may involve acquiringa first partially exposed image from an image sensor, acquiring a secondfurther exposed image from the image sensor, subsequently resetting theimage sensor. The first image may be obtained by subtracting the firstpartially exposed image from the second further exposed image. Thesecond image may be obtained from the image sensor according to certainembodiments immediately prior to the resetting.

In another alternative embodiment, a color filter enhancement method fora portable digital image acquisition device uses optics, a color sensorarray and a processor to acquire and process digital images. A firstrelatively underexposed and sharp image of a scene (“sharp image”) isobtained by exposing a first set of pixels of the sensor array. A secondrelatively well exposed and blurred image of the same scene (“blurredimage”) is acquired by exposing a second set of pixels of the sensorarray for a longer duration than the sharp image. The second set ofpixels in accordance with certain embodiments interleaves the first setof pixels. Color and/or luminance information may be applied from theblurred image to the sharp image, thereby constructing a color-enhancedand/or luminance-enhanced version of the sharp image.

The obtaining of the blurred image may include digitally exposing colorpixels of the sensor array for a first digital exposure duration. Theobtaining the sharp image may include digitally exposing white pixels ofthe sensor array for a second digital exposure time shorter than thefirst digital exposure duration. The constructing of the color-enhancedand/or luminance-enhanced version of the sharp image may include usingdata from both the color pixels exposed for the first digital exposureduration and the white pixels exposed for the second digital exposureduration. The digitally exposing of the color pixels and the whitepixels for different exposure times may include clocking the colorpixels and the white pixels independently. The digitally-exposing of thecolor pixels and the white pixels for different exposure times mayinvolve sensor data acquired over different and/or overlapping temporalranges, e.g., a first temporal range corresponding to thedigitally-exposing of the color pixels may include an entire secondtemporal range corresponding to the digitally-exposing of the whitepixels.

The color pixels may exhibit greater motion blurring effect than thewhite pixels due to the color pixels being digitally-exposed for alonger duration than the white pixels. The technique may involvecompensating blurring in the color pixels using less-blurred data fromthe white pixels.

In certain embodiments the second digital exposure time may include notmore than half of the first digital exposure time, or approximately athird of the first digital exposure time.

The color sensor array may include a CMOS-based sensor.

The technique may involve transmitting, storing and/or displaying thecolor-enhanced and/or luminance-enhanced version of the sharp image,and/or a further processed version.

In addition, in methods that may be performed according to preferred andalternative embodiments and claims herein, the operations have beendescribed in selected typographical sequences. However, the sequenceshave been selected and so ordered for typographical convenience and arenot intended to imply any particular order for performing theoperations, unless a particular ordering is expressly indicated as beingrequired or is understood by those skilled in the art as beingnecessary.

Many references have been cited above herein, and in addition to thatwhich is described as background, the invention summary, briefdescription of the drawings, the drawings and the abstract, thesereferences are hereby incorporated by reference into the detaileddescription of the preferred embodiments, as disclosing alternativeembodiments of elements or features of the preferred embodiments nototherwise set forth in detail above.

A single one or a combination of two or more of these references may beconsulted to obtain a variation of the preferred embodiments describedin the detailed description above. In addition, the following areincorporated by reference, particularly for this purpose: US2009/0303342, US2005/0041121, US2008/0043121, 2006/0204034,WO/2007/142621, U.S. Ser. No. 10/764,339, U.S. Pat. No. 7,369,712,US2005/0068452, US2006/0120599, 2006/0098237, US2006/0039690, U.S. Ser.No. 11/573,713, U.S. Ser. No. 12/042,335, US2007/0147820,US2007/0189748, US2009/0003652, and U.S. Pat. Nos. 7,336,821, 7,315,630,7,316,631, 7,403,643, and 7,460,695, and WO/2008/017343, US2010/0126831,US2007/0269108, US2007/0296833, US2008/0292193, US2008/0175481, U.S.Ser. No. 12/042,104, U.S. Ser. No. 12/330,719, U.S. Ser. No. 11/856,721,U.S. Ser. No. 12/026,484, U.S. Ser. No. 11/861,854, U.S. Ser. No.12/354,707, and U.S. Ser. No. 12/336,416. In addition, US publishedapplications nos. 2003/0169818, 20030193699, 20050041123, 20060170786,and 20070025714 are incorporated by reference, particularly asdisclosing alternative embodiments relating to still image camerasand/or the fourth embodiment “Dual image Acquisition Pipeline”.

What is claimed is:
 1. A method of generating sharp, well-exposed, colorimages from low-light images, comprising: acquiring a series ofunder-exposed images; computing a mean image based on an average ofmultiple images of the series of under-exposed images; generating a sumimage based on a sum of multiple images of the series of under-exposedimages; mapping chrominance variables of pixels of the mean image tochrominance variables of pixels of the sum image; replacing chrominancevalues of pixels within the series of under-exposed images withchrominance values of the sum image; and generating a set of sharp,well-exposed, color images based on the series of under-exposed imageswith replaced chrominance values.
 2. The method of claim 1, wherein theacquiring of said series of under-exposed images comprises capturing rawimages and generating processed RGB images based on the raw images. 3.The method of claim 2, wherein the computing of said mean imagecomprises applying a tone-mapping transformation to the series ofunder-exposed images.
 4. The method of claim 3, wherein the tone-mappingtransformation comprises a logarithmic transformation.
 5. The method ofclaim 2, wherein the acquiring of said series of under-exposed imagesfurther comprises applying a tone-mapping transformation to the seriesof under-exposed images.
 6. The method of claim 5, wherein thetone-mapping transformation comprises a logarithmic transformation. 7.The method of claim 1, wherein the acquiring of said series ofunder-exposed images comprises applying a tone-mapping transformation tothe series of under-exposed images.
 8. The method of claim 7, whereinthe tone-mapping transformation comprises a logarithmic transformation.9. The method of claim 1, further comprising transforming the mean andsum images to YUV space prior to mapping and replacing said chrominancevalues.
 10. The method of claim 9, further comprising transforming theunder-exposed images with replaced chrominance values back to RGB space.11. The method of claim 1, wherein the set of sharp, well-exposed, colorimages comprises multiple video images.
 12. The method of claim 1,wherein the set of sharp, well-exposed, color images comprise one ormore still images.
 13. A digital video acquisition device, comprising: alens and an image sensor for capturing a stream of multiple digitalvideo or still images; a processor; and a memory havingprocessor-readable code embedded therein for programming the processorto perform a method of generating sharp, well-exposed, color images fromunder-exposed, low-light images, wherein the method comprises: acquiringa series of under-exposed images; computing a mean image based on anaverage of multiple images of the series of under-exposed images;generating a sum image based on a sum of multiple images of the seriesof under-exposed images; mapping chrominance variables of pixels of themean image to chrominance variables of pixels of the sum image;replacing chrominance values of pixels within the series ofunder-exposed images with chrominance values of the sum image; andgenerating a set of sharp, well-exposed, color images based on theseries of under-exposed images with replaced chrominance values.
 14. Thedevice of claim 13, wherein the acquiring of said series ofunder-exposed images comprises capturing raw images and generatingprocessed RGB images based on the raw images.
 15. The device of claim14, wherein the computing of said mean image comprises applying atone-mapping transformation to the series of under-exposed images. 16.The device of claim 15, wherein the tone-mapping transformationcomprises a logarithmic transformation.
 17. The device of claim 14,wherein the acquiring of said series of under-exposed images furthercomprises applying a tone-mapping transformation to the series ofunder-exposed images.
 18. The device of claim 17, wherein thetone-mapping transformation comprises a logarithmic transformation. 19.The device of claim 13, wherein the acquiring of said series ofunder-exposed images comprises applying a tone-mapping transformation tothe series of under-exposed images.
 20. The device of claim 19, whereinthe tone-mapping transformation comprises a logarithmic transformation.21. The device of claim 13, wherein the method further comprisestransforming the mean and sum images to YUV space prior to mapping andreplacing said chrominance values.
 22. The device of claim 21, whereinthe method further comprises transforming the under-exposed images withreplaced chrominance values back to RGB space.
 23. The device of claim13, wherein the set of sharp, well-exposed, color images comprisesmultiple video images.
 24. The device of claim 13, wherein the set ofsharp, well-exposed, color images comprise one or more still images. 25.One or more non-transitory processor-readable media havingprocessor-readable code embedded therein for programming one or moreprocessors to perform a method of generating sharp, well-exposed, colorimages from under-exposed, low-light images, wherein the methodcomprises: acquiring a series of under-exposed images; computing a meanimage based on an average of multiple images of the series ofunder-exposed images; generating a sum image based on a sum of multipleimages of the series of under-exposed images; mapping chrominancevariables of pixels of the mean image to chrominance variables of pixelsof the sum image; replacing chrominance values of pixels within theseries of under-exposed images with chrominance values of the sum image;and generating a set of sharp, well-exposed, color images based on theseries of under-exposed images with replaced chrominance values.
 26. Theone or more non-transitory processor-readable media of claim 25, whereinthe acquiring of said series of under-exposed images comprises capturingraw images and generating processed RGB images based on the raw images.27. The one or more non-transitory processor-readable media of claim 26,wherein the computing of said mean image comprises applying atone-mapping transformation to the series of under-exposed images. 28.The one or more non-transitory processor-readable media of claim 27,wherein the tone-mapping transformation comprises a logarithmictransformation.
 29. The one or more non-transitory processor-readablemedia of claim 26, wherein the acquiring of said series of under-exposedimages further comprises applying a tone-mapping transformation to theseries of under-exposed images.
 30. The one or more non-transitoryprocessor-readable media of claim 29, wherein the tone-mappingtransformation comprises a logarithmic transformation.
 31. The one ormore non-transitory processor-readable media of claim 25, wherein theacquiring of said series of under-exposed images comprises applying atone-mapping transformation to the series of under-exposed images. 32.The one or more non-transitory processor-readable media of claim 31,wherein the tone-mapping transformation comprises a logarithmictransformation.
 33. The one or more non-transitory processor-readablemedia of claim 25, wherein the method further comprises transforming themean and sum images to YUV space prior to mapping and replacing saidchrominance values.
 34. The one or more non-transitoryprocessor-readable media of claim 33, wherein the method furthercomprises transforming the under-exposed images with replacedchrominance values back to RGB space.
 35. The one or more non-transitoryprocessor-readable media of claim 25, wherein the set of sharp,well-exposed, color images comprises multiple video images.
 36. The oneor more non-transitory processor-readable media of claim 25, wherein theset of sharp, well-exposed, color images comprise one or more stillimages.